A theory of word frequencies and its application to dialogue move recognition

Dialogue move recognition is taken as being representative of a class of spoken language applications where inference about high level semantic meaning is required from lower level acoustic, phonetic or word based features. Topic identication is another such application. In the particular case of inference from words, the multinomial distribution is shown to be inadequate for modelling word frequencies, and the multivariate Poisson is a more reasonable choice. Zipf's law is used to model a prior distribution. This more rigorous mathematical formulation is shown to improve dialogue move classi cation both subjectively and quantitatively.